From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less

Reproducing Kernel Hilbert Spaces (RKHS) have been found incredibly useful in the machine learning community. Their theory has been around for quite some time and has been used in the statistics literature for at least twenty years. More recently, their application to perceptron-style algorithms, as well as new classes of learning algorithms (specially large-margin or other regularization machines) has lead to the proliferation of algorithms and software that depend on their nature. Despite this burgeoning of practical uses, the theory and structure behind the use of the “kernel trick” is often glossed over. This tutorial attempts to take the reader from a very basic understanding of fields through Banach spaces and Hilbert spaces, into Reproducing Kernel Hilbert Spaces. This is very much a “RKHSs without the magic (with the math)” style paper, but every effort has been put in to making the math as clear as possible. For more information on the use of kernels in machine learning, the reader is referred to the well-known tutorials on support vector machines [3] and gaussian processes [9, 12]. Both SVMs and GPs belong to the class regularization learning machines that take advantage of the “kernel trick” to perform linear learning in non-linear spaces.